Skip to content

fix(authority): paginate canonical snapshots without mixing revisions - #4922

Merged
huangruiteng merged 6 commits into
mainfrom
codex/canonical-snapshot-pages-0923
Sep 24, 2026
Merged

huangruiteng merged 6 commits into
mainfrom
codex/canonical-snapshot-pages-0923

Conversation

@huangruiteng

@huangruiteng huangruiteng commented Sep 23, 2026 •

Copy link
Copy Markdown
Collaborator

Problem and result

A valid long-lived Goal could exceed the unchanged 2 MiB Python/TypeScript RPC response ceiling when a canonical read returned every Todo, archived dependency, lease and acceptance guard at once. The public list/status call then failed despite a healthy authority store. This PR assembles complete, byte-bounded pages tied to one provider incarnation, revision, cursor and query. An overlapping commit rejects the whole read; no partial collection or legacy display fallback escapes.

This is the TS T3 / shared-authority L5 read-transport slice for #4574. It does not change the default provider or promote an active Goal.

Implementation and semantics

  • TypeScript owns collection admission, page selection and acceptance guard membership. Both the existing one-shot endpoint and new page endpoint share the same Todo/read-model validation owner; Python only validates and assembles the transport.
  • Pages contain at most 4,096 full records and 1,792 KiB of UTF-8 JSON beneath the existing 2 MiB RPC ceiling. Ordinary collections stay on one RPC. A record or metadata object too large for a page fails explicitly.
  • Continuations bind Goal, store identity, provider revision, cursor, query and population. A changed head requires restarting the complete read, even if record content happens to look the same.
  • A fenced but absent File authority remains missing with the existing recovery response; reading never creates a replacement store. An existing head with an unreadable identity remains an identity failure. The public result schema also stays present on failure.
  • The complex synthetic production population now crosses the real byte ceiling while preserving native and imported shapes, archives, leases and acceptance guards. File, SQLite, NoKV transport and real PostgreSQL use the shared conformance contract.
  • The latest main introduced a bilingual RFC without the repository-required semantic-mirror declaration. A separate two-line docs commit repairs that pre-existing governance-smoke failure; it does not change this RFC's normative contract.

The bilingual RFC checkpoint lists seven principal PR boundaries including this one; after merge, six remain, or up to eight when SQLite qualification and migration each need a second reviewable batch. In-flight #4920 covers shadow-drain planning only; merged #4910 adds SQLite capacity axes but does not close the D2 soak/recovery gate. PostgreSQL service, tenant, restore and capacity qualification remains a separate medium-term lane.

Validation on 373ae0fa4add247abd316f9e4be25de21527156b

Check Result Evidence / limit
Python caller/recovery regression Passed 58 focused cases after the latest-main rebase, including split-root, missing authority and shadow fence.
Native page conformance Passed 26 File/qualified SQLite cases; the public-minimum Node 22.18 run passes 17 File cases and marks SQLite conformance skipped on its unqualified embedded driver.
Real PostgreSQL 16 Passed 288 provider integration cases against a separate disposable server/database, both removed after testing. The first local attempt used an old child Python and failed four import cases; the source-checkout interpreter rerun passed 288/288.
Real source and packaged entrypoint Passed Owner-authorized read-only Goal source copied to disposable File/SQLite authorities; complete readback and CLI with absent display passed, while source registry/state/leases/fences remained byte-identical. A rebuilt wheel matches all nine changed product files; extracted-wheel File/SQLite RPC and CLI readback passed on an over-budget synthetic collection. No active Goal was promoted.
Static and docs Passed TypeScript typecheck, configured mypy (23 modules), focused Ruff, generated-contract check, docs governance, public/private scan and diff hygiene.
Change-quality Passed Final 28-file scope fingerprint 33d86fc50fc36bf261c802933eaa4c40cbdb6b872dedd3c49ee5e6fdd2332e68; receipt cqr_33d86fc50fc36bf261c8 verified valid. No blockers or warnings; one efficiency advisory.
Risk-based premerge canary Passed 18 selected and 5 direct checks; no failures, warnings or manual holds.
GitHub CI Running Exact-head checks are queued/running; maintainer review and merge remain required.

Each page currently reloads the full head, and Python ultimately holds the complete collection. The RPC is bounded; total provider I/O and client memory are not. Busy writers may force an explicit full-read restart. This does not qualify SQLite D2 durability/soak, persistent projection freshness, whole-Goal migration or PostgreSQL deployment.

The CLI is the affected entrypoint. Dashboard, Chat and Lark continue to consume the same assembled public list/status contracts; there is no new setting or frontend asset. This control-plane PR stays open for maintainer review and merge.

Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
@huangruiteng
huangruiteng force-pushed the codex/canonical-snapshot-pages-0923 branch from a07903f to a5acb89 Compare September 23, 2026 05:58
… check

Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
Exact head: 373ae0f
English verdict: APPROVE — no blocking finding on this head.

动机:已验证旧的 one-shot canonical Todo/lease RPC 在完整集合超过 2 MiB 时失效;这会让已提升的长期 Goal 无法读取 Todo。此次交付是有用且独立可回滚的有界读传输增量,不宣称解决总体内存/IO 或后续 D1–D3。

改动思路:TypeScript 继续拥有 canonical Todo 选择和排序,复用同一 collection owner;新 page RPC 把 store identity、revision、cursor、query digest、数量与偏移绑定到 continuation,并限制每页字节和条数。Python 只校验、组装完整结果;任一页变更/异常就丢弃此前页面,不混合 revision、不回退 legacy。原 one-shot RPC 保留给既有直接调用者。

具体改动:检查了完整 base→head 28 文件差异、schema/handler/provider、Python promoted 与 pre-fence 分支、文档/RFC和相关调用者。源工作树验证:Python canonical snapshot 34 passed;TypeScript File/SQLite page 26 passed;旧路径/写入栅栏 72 passed;独立临时 PostgreSQL 16 数据库的完整 authority-store suite 288/288 passed(无 skip);control-plane typecheck 与 diff check 均通过。首次 PostgreSQL 运行的 4 个失败来自子进程误用系统旧 Python,改为工作树 uv 解释器后全套通过;不归因于此 PR。私有路径/凭据扫描无命中。

对主干的风险:最强反例是两页之间发生同内容或不同内容的 provider commit,导致表面合法的页面被拼成混合快照;负例覆盖了 revision/identity/offset 拒绝以及 Python 不返回部分结果。每页仍重读完整 head,最终 Python 结果仍占用总量内存,繁忙写入时可能反复重启;文档明确承认,这是后续按实测决定是否引入 pinned revision/streaming 的范围。没有对活动 Goal 做 promotion、迁移或原始数据改写,PostgreSQL 验证只用隔离的合成数据库。远端 CI 按本轮 review 契约不作为等待条件。

我的整体评价:未发现阻塞项。范围与已复现的 2 MiB 故障相称,规则仍归属于 TS 的 canonical owner,off/default legacy 分支有回归覆盖;可由 maintainer 合并。相关小型未来重构已通过共享 canonicalTodoCollection 完成,暂不扩展为无依据的流式框架。

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

动机

修复长生命周期 Goal 的完整 Todo/依赖/租约/验收守卫读取超过 2 MiB RPC 响应上限后,健康 authority store 却无法完成公共 list/status 的问题。本条是对同一 head 上先前 review 的格式修正;先前评论缺少能力要求的独立双语段落,不应作为有效 exact-head 结论。

改动思路

由 TypeScript authority 统一决定分页和成员资格,Python 只校验并组装传输;每页绑定 store identity、revision、cursor 和 query,读期间 revision 变化则拒绝整次读取,避免混合快照。

具体改动

在 373ae0fa4add247abd316f9e4be25de21527156b 上复核了分页读取、失败形态与前次 exact-head 验证记录。本轮重跑 34 项 Python 聚焦测试及 26 项 TypeScript 分页测试,均通过;此前同一 head 的独立 PostgreSQL 集成记录为 288/288,通过 source/packaged CLI、类型、文档与边界检查。没有修改此 PR 的代码。

对主干的风险

页间仍需重读 authority head,Python 最终仍持有完整集合;因此它解决 RPC 单响应上限,不解决总 I/O、内存或繁忙写入下的重试成本。并发 revision 变化会显式失败并要求从头读,这比悄然返回混合数据安全。未把本 PR 当作 SQLite 耐久性、完整迁移或 PostgreSQL 部署资格的完成证明。本轮未查询 CI;合并仍须 maintainer 按当前 head 判断。

我的整体评价

结论:认可当前 head 的这一有界传输修复,建议 maintainer 在所需检查与独立审阅满足后合并。相关的未来向重构已体现在单一 TypeScript 分页/成员资格 owner;暂不为未证实的 I/O 优化扩展本 PR。此评论取代此前格式无效的 review,且因作者身份只能作为 COMMENTED 自审结论,不能充当 GitHub formal approval。

English verdict: APPROVE - The exact head preserves a single authority-owned collection contract and rejects mixed-revision reads; focused revalidation passed. This is a review conclusion, not merge authorization.

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机

修复长生命周期 Goal 的完整 Todo/依赖/租约/验收守卫读取超过 2 MiB RPC 响应上限后,健康 authority store 却无法完成公共 list/status 的问题。本条是对同一 head 上先前 review 的格式修正;先前评论缺少能力要求的独立双语段落,不应作为有效 exact-head 结论。

改动思路

由 TypeScript authority 统一决定分页和成员资格,Python 只校验并组装传输;每页绑定 store identity、revision、cursor 和 query,读期间 revision 变化则拒绝整次读取,避免混合快照。

具体改动

在 373ae0fa4add247abd316f9e4be25de21527156b 上复核了分页读取、失败形态与前次 exact-head 验证记录。本轮重跑 34 项 Python 聚焦测试及 26 项 TypeScript 分页测试,均通过;此前同一 head 的独立 PostgreSQL 集成记录为 288/288,通过 source/packaged CLI、类型、文档与边界检查。没有修改此 PR 的代码。

对主干的风险

页间仍需重读 authority head,Python 最终仍持有完整集合;因此它解决 RPC 单响应上限,不解决总 I/O、内存或繁忙写入下的重试成本。并发 revision 变化会显式失败并要求从头读,这比悄然返回混合数据安全。未把本 PR 当作 SQLite 耐久性、完整迁移或 PostgreSQL 部署资格的完成证明。本轮未查询 CI;合并仍须 maintainer 按当前 head 判断。

我的整体评价

结论:认可当前 head 的这一有界传输修复,建议 maintainer 在所需检查与独立审阅满足后合并。相关的未来向重构已体现在单一 TypeScript 分页/成员资格 owner;暂不为未证实的 I/O 优化扩展本 PR。此评论取代此前格式无效的 review,且因作者身份只能作为 COMMENTED 自审结论,不能充当 GitHub formal approval。

English verdict: APPROVE - The exact head preserves a single authority-owned collection contract and rejects mixed-revision reads; focused revalidation passed. This is a review conclusion, not merge authorization.

Keep both reads added at the same seam: this branch's shared canonicalTodoCollection admission and main's exact operation-receipt readback. Take main's already-repaired bilingual mirror declaration for automatic-execution-admission-v0 so that governance repair is no longer duplicated.

Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机

长生命周期 Goal 的完整 canonical 读取(全部 Todo、归档依赖、租约与验收守卫)会超过未改动的 2 MiB Python/TypeScript RPC 响应上限,使公共 list/status 在 authority store 健康的情况下直接失败。本次交付是有用且可独立回滚的有界读传输增量:为 TS T3/shared-authority L5 的读传输切片,不改变默认 provider、不晋升任何活动 Goal,也不宣称解决总体 I/O 或内存开销。

本次是本 head aba1bd7a0bf102acf98de0152b69b51d0b4cf8b2 的重新评审:相对上一轮已评审的 373ae0fa4,本 head 只加入与最新 main 的合并(保留双方在 local_authority_read.ts 同一位置新增的不同读取函数,并采用 main 已修好的 automatic-execution-admission-v0 中英镜像声明,因此本分支不再重复该治理修复)。PR 自身的产品改动未变。

改动思路

TypeScript authority 继续拥有集合准入、分页选择与验收守卫成员资格:one-shot 端点与新的 page 端点共用同一个 canonicalTodoCollection,分页只改变传输。continuation 把 Goal、store identity、provider revision、cursor、query digest 与 population(todo_count/lease_count)绑定在一起,并要求严格向前推进(lease_offset > 0 时必须已读满 todo),任一页身份、排序或 revision 不符即整体拒绝,不返回部分集合、也不回退 legacy 显示。Python 只做校验与组装,读取使用 existingOnly 打开 authority,因此 fenced 但缺失的 store 仍是既有 missing 恢复语义,读不会创建替代存储。

具体改动

26 文件 +1374/−115。Python:local_authority.py 的 promoted 读取改走新的 read_canonical_snapshot;新增 canonical_snapshot.py 组装器(逐页校验、整体失败)。TypeScript:新增 canonical_snapshot_page.ts(每页 1792 KiB/4096 条上限、continuation 编解码);local_authority_read.ts 抽出共享准入并保留原 one-shot 路径;provider 增加 existingOnly 选项;handler 注册 coordination.local_authority.todo_snapshot_page。契约:coordination_state_contract_v0.json 与双边生成绑定、生成脚本、pyproject.toml 的 mypy 列表同步。测试:Python 分页/集成/契约与调用者用例、TS 分页与 provider conformance(File、SQLite、NoKV、真实 PostgreSQL 共用同一 conformance 契约)。文档:新增 canonical snapshot pagination 参考并更新中英 RFC 与执行台账。

关键代码讲解

  • canonicalTodoCollection(loopx/control_plane/coordination/local_authority_read.ts):one-shot 与分页共用的唯一准入与成员资格所有者,分页不引入第二套 Todo/读模型语义。
  • readCanonicalSnapshotPage 与 decodePosition(loopx/control_plane/coordination/canonical_snapshot_page.ts):只接受精确字段的 continuation,校验 todo_offset/lease_offset 的推进与顺序约束,并让每页落在字节与条数预算内。
  • read_canonical_snapshot(loopx/control_plane/coordination/canonical_snapshot.py):逐页拉取并校验后组装完整结果;任一页失败即整体失败,公共结果 schema 在失败时仍然存在。
  • provider existingOnly(loopx/control_plane/coordination/local_authority_provider.ts):把"读不得创建 store"从调用约定变成类型内的开关。
  • 本 head 的合并解决:canonicalTodoCollection 与 main 的 readLocalCoordinationOperationReceipt 在同一 import 后并存,两条路由分别保留在 handler 表中。

对主干的风险

最强反例是两页之间发生 provider commit,使表面合法的页面被拼成混合快照;负例覆盖 revision、store identity、offset 与 population 不符时的整体拒绝,以及 Python 不返回部分结果。已知限制(文档已声明):每页仍重读完整 head,Python 最终仍持有完整集合,因此本 PR 限定的是 RPC 单响应上限,而不是总 provider I/O、客户端内存或繁忙写入下的重启成本;并发 revision 变化会显式失败并要求从头读取,这比静默返回混合数据安全。

验证矩阵(本轮在 aba1bd7a0 上执行):Python 契约生成检查、canonical snapshot、集成与 project-registry census 共 61 用例通过;promoted 调用方、todo 投影并发与 legacy writer fence 共 63 用例通过;TS 分页与 conformance 定向 27 用例通过;TS 全量 npm run test:control-plane 2931/2956 通过,唯一失败 tests/control_plane_ts/sqlite_capacity.test.ts 的 rehearsal 入口在未包含本 PR 的 origin/main 上以同样方式失败,属于既有环境问题;typecheck:control-plane、配置范围内 Ruff、python -m mypy(23 模块)与 examples/docs-governance-smoke.py 通过。按本 Goal 的 wait_for_ci=false 与维护者本轮要求,未抓取或等待远端 CI;合并前仍对未变 head 运行 merge-readiness。未把本 PR 当作 SQLite D2 耐久性/soak、整 Goal 迁移或 PostgreSQL 部署资格的完成证明。

语义与 CI 对齐

新增的 request/result schema 是既有 coordination 契约的扩展,通过仓库生成器产出 Python/TypeScript 双边绑定,并纳入既有契约一致性检查;持久 schema、公共 CLI 参数与 provider 默认行为未变。

我的整体评价

APPROVE:范围与已复现的 2 MiB 故障相称,规则仍归属既有 TS canonical owner,off/legacy 分支与失败形态都有回归覆盖;本 head 相对上一轮评审只增加 main 合并,未新增产品改动。相关的未来向重构已体现在单一分页/成员资格 owner,暂不为未证实的 I/O 优化扩展本 PR。剩余本地 authority 默认交付边界、SQLite D2 与 PostgreSQL 部署资格仍按现有 RFC 条件路径推进;本批准不等于 provider 晋升或控制面自合并许可。

English verdict: APPROVE - exact head aba1bd7a0bf102acf98de0152b69b51d0b4cf8b2 keeps one authority-owned collection admission behind both the one-shot and byte-bounded paged reads, rejects mixed-revision or out-of-order continuations, and merges latest main without changing the reviewed product delta; 124 focused Python cases, 27 targeted TypeScript cases, the full TypeScript suite minus one failure that reproduces on unmodified main, typecheck, Ruff, mypy and the docs governance smoke passed locally.

@huangruiteng
huangruiteng merged commit bf0df7b into main Sep 24, 2026
8 checks passed
@huangruiteng
huangruiteng deleted the codex/canonical-snapshot-pages-0923 branch September 24, 2026 04:35
@huangruiteng

Copy link
Copy Markdown
Collaborator Author

Merged decision record (admin-bypass merge, maintainer-authorized).

  • Exact head reviewed and merged: aba1bd7a0bf102acf98de0152b69b51d0b4cf8b2; merge commit on main: bf0df7b4131a822e5b1dae56eac51be7e5a5043a.
  • Before the merge, loopx pr-review --check-merge-readiness 4922@aba1bd7a0... returned ready=true with no blocking reasons, using the author-owned exact-head review published above.
  • git diff aba1bd7a0... bf0df7b41 is empty: the merged main tree is byte-identical to the head that was validated.

Change relative to the previously reviewed head 373ae0fa4: merge of latest main only. The conflict in local_authority_read.ts was resolved by keeping both functions added at the same seam (this branch's canonicalTodoCollection admission and main's readLocalCoordinationOperationReceipt), and the automatic-execution-admission-v0 mirror declaration now takes main's already-repaired wording, so this branch no longer duplicates that governance fix. No product change was added beyond the merge.

Validation at the merged head:

  • Passed: 61 Python contract/canonical-snapshot/integration/census cases; 63 promoted-caller, projection-concurrency and writer-fence cases; 27 targeted TypeScript page/conformance cases; typecheck:control-plane; configured Ruff; mypy (23 modules); examples/docs-governance-smoke.py; change-quality receipt cqr_11871be398c743f84796 (status=valid).
  • TypeScript full suite: 2931/2956. The single failure, tests/control_plane_ts/sqlite_capacity.test.ts ("small capacity entrypoint exercises real SQLite..."), reproduces identically on unmodified origin/main and is unrelated to this diff.
  • Not awaited: remote CI, per this Goal's wait_for_ci=false and the maintainer's explicit instruction for this run. CI on the previous head 373ae0fa4 was green apart from the advisory node-forward-compatibility lane.
  • Known limits carried by this PR: the RPC response is bounded, but each page reloads the authority head and Python still holds the complete collection, so total I/O, client memory and busy-writer restart cost remain open and are documented in docs/reference/canonical-snapshot-pagination.md. This does not qualify SQLite D2 durability/soak or PostgreSQL deployment.

huangruiteng added a commit that referenced this pull request Sep 24, 2026
…readmodel-0924

Two conflicts. The effect-runtime handler registry conflicted on imports
only, so both sides keep their handlers. The shared-authority RFC conflicted
on the delivery-count checkpoint: main carries the 2026-09-23 seven-boundary
decomposition while this branch recorded its own dependency reconciliation.
Resolution keeps main's decomposition, updates the reconciliation with the
now-merged #4922/#4960/#4961 and the still-in-review #4931, and states that the
estimate is updated after the combined head is accepted instead of restoring
the superseded five-to-eight package range.

Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant